Communication of Information Regarding a Robot Using an Optical Identifier

ABSTRACT

A control system may perform functions including (i) storing data indicating an association between an optical identifier and a first robot, (ii) sending, to the first robot, data encoding the optical identifier for display by the first robot, and (iii) after sending the data encoding the optical identifier, sending, to a second robot, the data indicating the association between the optical identifier and the first robot. In some examples, the first robot may receive, from the control system, data encoding a second optical identifier of the first robot so that the first robot may display the second optical identifier instead of the first optical identifier. In some examples, a first robot may capture an image of an indication of a priority status of a second robot and perform an action based on comparing a first priority status of the first robot to the second priority status of the second robot.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. §120 to, and is acontinuation of, U.S. patent application Ser. No. 14/922,738, filed onOct. 26, 2015, entitled “Communication of Information Regarding a RobotUsing an Optical Identifier,” which is incorporated herein by referencein its entirety.

BACKGROUND

A warehouse may be used for storage of goods by a variety of differenttypes of commercial entities, including manufacturers, wholesalers, andtransport businesses. Example stored goods may include raw materials,parts or components, packing materials, and finished products. In somecases, the warehouse may be equipped with loading docks to allow goodsto be loaded onto and unloaded from delivery trucks or other types ofvehicles. The warehouse may also include rows of pallet racks forstorage of pallets, and flat transport structures that contain stacks ofboxes or other objects. Additionally, the warehouse may have machines orvehicles for lifting and moving goods or pallets of goods, such ascranes and forklifts. Human operators may be employed to operatemachines, vehicles, and other equipment within the warehouse. In somecases, one or more of the machines or vehicles may be robotic devicesguided by computer control systems.

SUMMARY

In one example, a control system includes one or more processors and acomputer-readable storage medium storing instructions that, whenexecuted by the one or more processors, cause the control system toperform functions. The functions include storing data indicating anassociation between an optical identifier and a first robot, sending, tothe first robot, data encoding the optical identifier for display by thefirst robot, and after sending the data encoding the optical identifier,sending, to a second robot, the data indicating the association betweenthe optical identifier and the first robot.

In another example, a computer-readable medium stores instructions that,when executed by a control system, cause the control system to performfunctions. The functions include storing data indicating an associationbetween an optical identifier and a first robot, sending, to the firstrobot, data encoding the optical identifier for display by the firstrobot, and after sending the data encoding the optical identifier,sending, to a second robot, the data indicating the association betweenthe optical identifier and the first robot.

In yet another example, a method includes storing data indicating anassociation between an optical identifier and a first robot, sending, tothe first robot, data encoding the optical identifier for display by thefirst robot, and after sending the data encoding the optical identifier,sending, to a second robot, the data indicating the association betweenthe optical identifier and the first robot.

In yet another example, a control system includes means for storing dataindicating an association between an optical identifier and a firstrobot, means for sending, to the first robot, data encoding the opticalidentifier for display by the first robot, and means for after sendingthe data encoding the optical identifier, sending, to a second robot,the data indicating the association between the optical identifier andthe first robot.

In yet another example, a robot includes one or more processors, anoptical communication interface, and a computer-readable storage mediumstoring instructions that, when executed by the one or more processors,cause the robot to perform functions. The functions include receivingdata encoding a first optical identifier of the robot, displaying, bythe optical communication interface, the first optical identifier,receiving data encoding a second optical identifier of the robot, anddisplaying, by the optical communication interface, the second opticalidentifier.

In yet another example, a computer-readable storage medium storesinstructions that, when executed by a robot comprising an opticalcommunication interface, cause the robot to perform functions. Thefunctions include receiving data encoding a first optical identifier ofthe robot, displaying, by the optical communication interface, the firstoptical identifier, receiving data encoding a second optical identifierof the robot, and displaying, by the optical communication interface,the second optical identifier.

In yet another example, a method performed by a robot comprising anoptical communication interface includes receiving data encoding a firstoptical identifier of the robot, displaying, by the opticalcommunication interface, the first optical identifier, receiving dataencoding a second optical identifier of the robot, and displaying, bythe optical communication interface, the second optical identifier.

In yet another example, a robot includes means for receiving dataencoding a first optical identifier of the robot, means for displayingthe first optical identifier, means for receiving data encoding a secondoptical identifier of the robot, and means for displaying the secondoptical identifier.

In yet another example, a first robot includes one or more processorsand a computer-readable storage medium storing instructions that, whenexecuted by the one or more processors, cause the first robot to performfunctions. The functions include receiving, from a control system, dataencoding a first priority status of the first robot and capturing animage of an indication of a second priority status of a second robot.The indication is displayed by the second robot. The functions furtherinclude determining the second priority status by identifying theindication within the captured image, comparing the first prioritystatus to the second priority status, and performing an action based oncomparing the first priority status to the second priority status.

In yet another example, a computer-readable medium stores instructionsthat, when executed by a first robot, cause the first robot to performfunctions. The functions include receiving, from a control system, dataencoding a first priority status of the first robot and capturing animage of an indication of a second priority status of a second robot.The indication is displayed by the second robot. The functions furtherinclude determining the second priority status by identifying theindication within the captured image, comparing the first prioritystatus to the second priority status, and performing an action based oncomparing the first priority status to the second priority status.

In yet another example, a method performed by a first robot includesreceiving, from a control system, data encoding a first priority statusof the first robot and capturing an image of an indication of a secondpriority status of a second robot. The indication is displayed by thesecond robot. The method further includes determining the secondpriority status by identifying the indication within the captured image,comparing the first priority status to the second priority status, andperforming an action based on comparing the first priority status to thesecond priority status.

In yet another example, a first robot includes means for receiving, froma control system, data encoding a first priority status of the firstrobot and means for capturing an image of an indication of a secondpriority status of a second robot. The indication is displayed by thesecond robot. The first robot further includes means for determining thesecond priority status by identifying the indication within the capturedimage, means for comparing the first priority status to the secondpriority status, and means for performing an action based on comparingthe first priority status to the second priority status.

These as well as other aspects, advantages, and alternatives, willbecome apparent to those of ordinary skill in the art by reading thefollowing detailed description, with reference where appropriate to theaccompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows a robotic fleet, according to an example embodiment.

FIG. 1B is a functional block diagram illustrating components of arobotic fleet, according to an example embodiment.

FIG. 2A shows a truck (un)loader, according to an example embodiment.

FIG. 2B shows a pedestal robot, according to an example embodiment.

FIG. 2C shows an autonomous guided vehicle, according to an exampleembodiment.

FIG. 2D shows an autonomous fork truck, according to an exampleembodiment.

FIG. 2E shows a truck (un)loader, according to an example embodiment.

FIGS. 3A, 3B, 3C, 3D, and 3E illustrate example operations of a roboticfleet in a warehouse, according to an example embodiment.

FIG. 4 is a block diagram of a method, according to an exampleembodiment.

FIG. 5 is a block diagram of another method, according to an exampleembodiment.

FIG. 6 is a block diagram of yet another method, according to an exampleembodiment.

FIG. 7 shows an example autonomous fork truck and an example pallet,according to an example embodiment.

FIG. 8 shows example truck (un)loaders and an example delivery truck,according to an example embodiment.

DETAILED DESCRIPTION

Example methods and systems are described herein. Any exampleimplementation or feature described herein is not necessarily to beconstrued as preferred or advantageous over other implementations orfeatures. The example implementations described herein are not meant tobe limiting. Certain aspects of the disclosed systems and methods can bearranged and combined in a wide variety of different configurations, allof which are contemplated herein.

Furthermore, the particular arrangements shown in the Figures should notbe viewed as limiting. It should be understood that other embodimentsmight include more or fewer of each element shown in a given Figure.Further, some of the illustrated elements may be combined or omitted.Yet further, an example embodiment may include elements that are notillustrated in the Figures.

Multiple robots may act in coordination to perform a variety of tasks,perhaps at the direction of a control system. In one such example, thecontrol system may coordinate the robots as the robots move items withina warehouse. In this context, the control system may monitor andidentify various robots within the warehouse to observe and aid in tasksbeing performed by the robots. One way for the control system toidentify a given robot in the warehouse might be to use a camera oranother optical sensor to detect and identify a “static” identifier(e.g., a two-dimensional matrix code) that appears on the exterior ofthe given robot. However, static identifiers can be vulnerable to“spoofing.” For example, a malicious party may deploy within thewarehouse an imposter robot marked with another robot's identifier. Themalicious party may then use the imposter robot to disrupt or hijack theoperations of the other robots by “tricking” the other robots intoacting in coordination with the imposter robot.

To address this problem, the robots and/or the control system may use“dynamic” optical identifiers that are reassigned to respective robotsperiodically or upon detection of various events. For example, thecontrol system may send to a first robot, data encoding an opticalidentifier of the first robot. The optical identifier may include anysort of fiducial marker, such as a shape, pattern, marking, ortwo-dimensional matrix code recognizable as corresponding to the firstrobot. The optical identifier may be displayed on a display screen ofthe first robot. In another example, the first robot may use a lightsource such as a light-emitting diode (LED) to display a series offlashing visible or infrared pulses as the optical identifier.Thereafter, the control system may periodically, or upon identificationof a potential security breach of the warehouse, send to the first robotnew data encoding a new optical identifier of the first robot. The firstrobot may then display the new optical identifier for detection by otherrobots or the control system.

The control system may maintain a database of optical identifiersrespectively assigned to various robots in the warehouse. Such adatabase may be accessible by any or all of the robots in the warehouse.The control system may detect the first robot's optical identifier andaccess the database to associate the detected optical identifier withthe first robot. The control system may then observe the first robot todetermine a state of the first robot, including one or more of a currentlocation of the first robot, the identity of an item or another robotthat the first robot is interacting with, or an operational status etc.Based on the state of the first robot and in accordance with overallsystem goals, the control system may send a command to the first robotto perform a particular function.

In another example, a second robot may detect the first robot's opticalidentifier and send a request to the control system for informationidentifying the first robot. Such a request may include data encodingthe optical identifier that is displayed by the first robot and detectedby the second robot. In reply, the control system may send to the secondrobot a network address of the first robot. In some examples, the secondrobot may send a message or a command to the network address of thefirst robot. Such a message or command may cause the first robot toperform functions that further the overall system goals.

In addition to identifying the first robot, the optical identifier ofthe first robot may indicate a priority status of the first robot. Forexample, the first robot may be tasked with bringing a first item to athird robot and the second robot may be tasked with bringing a seconditem to the third robot. If both the first and second robots approachthe third robot at about the same time, the first and second robots mayneed to determine which of the first and second robots will bring theirrespective item to the third robot first. For example, the opticalidentifier of the first robot may indicate a priority status “1” whereasthe optical identifier of the second robot may indicate a prioritystatus “2.” In this case, the first robot may detect the opticalidentifier of the second robot, determine that the priority status ofthe first robot is higher than the priority status of the second robot,and proceed to bring the first item to the third robot without yieldingto the second robot. Similarly, the second robot may detect the opticalidentifier of the first robot, determine that the priority status of thefirst robot is higher than the priority status of the second robot, andproceed to bring the second item to the third robot only afterdetermining that the first robot has departed the vicinity of the thirdrobot.

Reference will now be made in detail to various embodiments, examples ofwhich are illustrated in the accompanying drawings. In the followingdetailed description, numerous specific details are set forth in orderto provide a thorough understanding of the present disclosure and thedescribed embodiments. However, the present disclosure may be practicedwithout these specific details. In other instances, well-known methods,procedures, components, and circuits have not been described in detailso as not to unnecessarily obscure aspects of the embodiments.

37 Example embodiments may involve a robotic fleet deployed within awarehouse environment. More specifically, a combination of fixed andmobile components may be deployed within the environment to facilitateautomated processing of boxes, packages, or other types of objects.Example systems may involve automated loading and/or unloading of boxesand/or other objects, such as into storage containers or to and fromdelivery vehicles. In some example embodiments, boxes or objects may beautomatically organized and placed onto pallets. Within examples,automating the process of loading/unloading trucks and/or the process ofcreating pallets from objects for easier storage within a warehouseand/or for transport to and from the warehouse may provide a number ofindustrial and business advantages.

According to various embodiments, automating the process ofloading/unloading delivery trucks at the warehouse and/or the process ofcreating pallets may include the deployment of one or more differenttypes of robotic devices to move objects or perform other functions. Insome embodiments, some of the robotic devices can be made mobile bycoupling with a wheeled base, a holonomic base (e.g., a base that canmove in any direction), or rails on the ceiling, walls or floors. Inadditional embodiments, some of the robotic devices may be made fixedwithin the environment as well. For instance, robotic manipulators canbe positioned on elevated bases at different chosen locations within awarehouse.

As used herein, the term “warehouse” may refer to any physicalenvironment in which boxes or objects may be manipulated, processed,and/or stored by robotic devices. In some examples, a warehouse may be asingle physical building or structure, which may additionally containcertain fixed components, such as pallet racks for storing pallets ofobjects. In other examples, some fixed components may be installed orotherwise positioned within the environment before or during objectprocessing. In additional examples, a warehouse may include multipleseparate physical structures, and/or may also include physical spacesthat are not covered by a physical structure as well.

Further, the term “boxes” may refer to any object or item that can beplaced onto a pallet or loaded onto or unloaded from a truck orcontainer. For example, in addition to rectangular solids, “boxes” canrefer to cans, drums, tires or any other “simple” geometric items.Additionally, “boxes” may refer to totes, bins, or other types ofcontainers which may contain one or more items for transport or storage.For instance, plastic storage totes, fiberglass trays, or steel bins maybe moved or otherwise manipulated by robots within a warehouse. Examplesherein may also be applied toward objects other than boxes as well, andtoward objects of various sizes and shapes. Additionally, “loading” and“unloading” can each be used to imply the other. For instance, if anexample describes a method for loading a truck, it is to be understoodthat substantially the same method can also be used for unloading thetruck as well. As used herein, “palletizing” refers to loading boxesonto a pallet and stacking or arranging the boxes in a way such that theboxes on the pallet can be stored or transported on the pallet. Inaddition, the terms “palletizing” and “depalletizing” can each be usedto imply the other.

Within examples, a heterogeneous warehouse robot fleet may be used for anumber of different applications. One possible application includesorder fulfillment (e.g., for individual customers), in which cases maybe opened and individual items from the cases may be put into packagingwithin boxes to fulfill individual orders. Another possible applicationincludes distribution (e.g., to stores or other warehouses), in whichmixed pallets may be constructed containing groups of different types ofproducts to ship to stores. A further possible application includescross-docking, which may involve transporting between shippingcontainers without storing anything (e.g., items may be moved from four40-foot trailers and loaded into three lighter tractor trailers, andcould also be palletized). Numerous other applications are alsopossible.

Referring now to the figures, FIG. 1A depicts a robotic fleet within awarehouse setting, according to an example embodiment. Morespecifically, different types of robotic devices may form aheterogeneous robotic fleet 100 that may be controlled to collaborate toperform tasks related to the processing of items, objects, or boxeswithin a warehouse environment. Certain example types and numbers ofdifferent robotic devices are shown here for illustration purposes, butrobotic fleet 100 may employ more or fewer robotic devices, may omitcertain types shown here, and may also include other types of roboticdevices not explicitly shown. Additionally, a warehouse environment isshown here with certain types of fixed components and structures, butother types, numbers, and placements of fixed components and structuresmay be used in other examples as well.

The robotic fleet 100 may include various types of mobile vehicles. Oneexample type of robotic device shown within robotic fleet 100 is anautonomous guided vehicle (AGV) 112, which may be a relatively small,mobile device with wheels that may function to transport individualpackages, cases, or totes from one location to another within thewarehouse. Another example type of robotic device is an autonomous forktruck 114, a mobile device with a forklift that may be used to transportpallets of boxes and/or to lift pallets of boxes (e.g., to place thepallets onto a rack for storage). An additional example type of roboticdevice is a robotic truck loader/unloader 116, a mobile device with arobotic manipulator as well as other components such as optical sensorsto facilitate loading and/or unloading boxes onto and/or off of trucksor other vehicles. For instance, robotic truck unloader 116 may be usedto load boxes onto delivery truck 118, which may be parked adjacent tothe warehouse. In some examples, movements of delivery truck 118 (e.g.,to deliver packages to another warehouse) may also be coordinated withrobotic devices within the fleet.

Various types of robotic devices other than those illustrated here maybe included in an example system. In some examples, one or more roboticdevices may use alternative modes of locomotion instead of wheel-basedlocomotion. For instance, one or more robotic devices may berotor-driven so as to operate airborne. For instance, an airbornerobotic device such as a quadcopter may be used for various tasks suchas moving objects or collecting sensor data.

In further examples, the robotic fleet 100 may also include variousfixed components that may be positioned within the warehouse. In someexamples, one or more fixed robotic devices may be used to move orotherwise process boxes. For instance, a pedestal robot 122 may includea robotic arm elevated on a pedestal that is fixed to the ground floorwithin the warehouse. The pedestal robot 122 may be controlled todistribute boxes between other robots and/or to stack and unstackpallets of boxes. For example, the pedestal robot 122 may pick up andmove boxes from nearby pallets 140 and distribute the boxes toindividual AGV's 112 for transportation to other locations within thewarehouse.

In additional examples, robotic fleet 100 may employ additional fixedcomponents positioned within a warehouse space. For instance, highdensity storage 124 may be used to store pallets and/or objects withinthe warehouse. The density storage 124 may be designed and positioned tofacilitate interaction with one or more robotic devices within thefleet, such as autonomous fork truck 114. In further examples, certainground space may be selected and used for storage of pallets or boxes aswell or instead. For instance, pallets 130 may be positioned within thewarehouse environment at chosen locations for certain periods of time toallow the pallets to be picked up, distributed, or otherwise processedby one or more of the robotic devices.

FIG. 1B is a functional block diagram illustrating components of arobotic fleet 100, according to an example embodiment. The robotic fleet100 could include one or more of various mobile components 110, such asAGV's 112, autonomous fork trucks 114, robotic truck loaders/unloaders116, and delivery trucks 118. The robotic fleet 100 may additionallyinclude one or more fixed components positioned within a warehouse orother environment, such as pedestal robots 122, density storage 124, andbattery exchange/charging stations 126. In further examples, differentnumbers and types of the components illustrated within FIG. 1B may beincluded within a fleet, certain types may be omitted, and additionalfunctional and/or physical components may be added to the examplesillustrated by FIG. 1A and 1B as well. To coordinate actions of separatecomponents, a control system 150, such as a remote, cloud-based serversystem, may communicate (e.g., by way of wireless communicationinterfaces) with some or all of the system components and/or withseparate local control systems of individual components.

Any of the mobile components 110 may include one or more processors 113a and a non-transitory computer-readable storage medium 113 b thatstores instructions executable by the one or more processors 113 a toperform any function or action described herein. The mobile components110 may also each include a wireless communication interface (e.g.,WIFI, Bluetooth, etc.) so that the mobile components 110 may transmitdata to and/or receive data from any of the other mobile components 110,the pedestal robots 122, and/or the control system 150.

Within examples, certain of the fixed components 120 may be installedbefore deployment of the rest of the robotic fleet 100. In someexamples, one or more mobile robots may be brought in to map a spacebefore determining placement of certain fixed components 120, such asthe pedestal robots 122 or battery exchange stations 126. Once mapinformation is available, the system may determine (e.g., by runningsimulations) how to layout the fixed components within the spaceavailable. In certain cases, a layout may be chosen to minimize thenumber of fixed components needed and/or the amount of space used bythose components. The fixed components 120 and mobile components 110 maybe deployed in separate stages or all at once. In additional examples,certain of the mobile components 110 may only be brought in duringparticular time periods or to complete particular tasks.

Any of the pedestal robots 122 may include one or more processors 123 aand a non-transitory computer-readable storage medium 123 b that storesinstructions executable by the one or more processors 123 a to performany function or action described herein. The pedestal bots 122 may alsoeach include a wireless communication interface (e.g., WIFI, Bluetooth,etc.) so that the pedestal bots 122 may transmit data to and/or receivedata from the control system 150, other pedestal robots 122, and/or anyof the mobile components 110.

In some examples, control system 150 may include a central planningsystem that assigns tasks to different robotic devices within fleet 100.The central planning system may employ various scheduling algorithms todetermine which devices will complete which tasks at which times. Forinstance, an auction type system may be used in which individual robotsbid on different tasks, and the central planning system may assign tasksto robots to minimize overall costs. In additional examples, the centralplanning system may optimize across one or more different resources,such as time, space, or energy utilization. In further examples, aplanning or scheduling system may also incorporate particular aspects ofthe geometry and physics of box picking, packing, or storing.

The control system 150 may include one or more processors 151 and anon-transitory computer-readable storage medium 152 that storesinstructions executable by the one or more processors 151 for executingany function or action described herein. The control system 150 may alsoinclude a wireless communication interface (e.g., WIFI, Bluetooth, etc.)so that the control system 150 may transmit data to and/or receive datafrom the any of the mobile components 110 and/or the pedestal robots122. The control system 150 may also include a camera (or becommunicatively coupled to a camera) for capturing images of theenvironment of the control system 150.

Planning control may also be distributed across individual systemcomponents. For example, control system 150 may issue instructionsaccording to a global system plan, and individual system components mayalso operate according to separate local plans. Additionally, differentlevels of detail may be included within a global plan, with otheraspects left for individual robotic devices to plan locally. Forinstance, mobile robotic devices may be assigned target destinations bya global planner but the full routes to reach those target destinationsmay be planned or modified locally.

In additional examples, a central planning system may be used inconjunction with local vision on individual robotic devices tocoordinate functions of robots within robotic fleet 100. For instance, acentral planning system may be used to get robots relatively close towhere they need to go. However, it may be difficult for the centralplanning system to command robots with millimeter precision, unless therobots are bolted to rails or other measured components are used toprecisely control robot positions. Local vision and planning forindividual robotic devices may therefore be used to allow for elasticitybetween different robotic devices. A general planner may be used to geta robot close to a target location, at which point local vision of therobot may take over. In some examples, most robotic functions may beposition-controlled to get the robots relatively close to targetlocations, and then vision and handshakes may be used when needed forlocal control.

In further examples, visual handshakes may enable two robots to identifyone another by AR tag or other characteristics, and to performcollaborative operations within fleet 100. In additional examples, items(e.g., packages to be shipped) may be provided with visual tags as wellor instead, which may be used by robotic devices to perform operationson the items using local vision control. In particular, the tags may beused to facilitate manipulation of the items by the robotic devices. Forinstance, one or more tags on particular locations on a pallet may beused to inform a fork lift where or how to lift up the pallet.

In additional examples, deployment and/or planning strategies for fixedand/or mobile components may be optimized over time. For instance, acloud-based server system may incorporate data and information fromindividual robots within the fleet and/or from external sources.Strategies may then be refined over time to enable the fleet to use lessspace, less time, less power, less electricity, or to optimize acrossother variables. In some examples, optimizations may span acrossmultiple warehouses, possibly including other warehouses with roboticfleets and/or traditional warehouses. For instance, control system 150may incorporate information about delivery vehicles and transit timesbetween facilities into central planning.

In some examples, a central planning system may sometimes fail, such aswhen a robot gets stuck or when packages get dropped in a location andlost. Local robot vision may also therefore provide robustness byinserting redundancy to handle cases where the central planner fails.For instance, as an automatic pallet jack passes and identifies anobject, the pallet jack may send information up to a remote, cloud-basedserver system. Such information may be used to fix errors in centralplanning, help to localize robotic devices, or to identify lost objects.

In further examples, a central planning system may maintain or haveaccess to a map of the physical environment containing robotic fleet 100and objects undergoing processing by the robotic devices. In someexamples, the map may be continuously updated with information aboutdynamic objects (e.g., moving robots and packages moved by robots). Inadditional examples, a dynamic map could include information about boththe current configuration or placement of components within a warehouse(or across multiple warehouses) as well as information about theconfiguration or placement of components that is anticipated in the nearterm. For instance, the map could show current locations of movingrobots and anticipated locations of the robots in the future, which maybe used to coordinate activity between robots. The map could also showcurrent locations of items undergoing processing as well as anticipatedfuture locations of the items (e.g., where an item is now and when theitem is anticipated to be shipped out).

In additional examples, some or all of the robots may scan for labels onobjects at different points within the process. The scans may be used tolook for visual tags that may be applied to individual components orspecific items to facilitate finding or keeping track of components anditems. This scanning may yield a trail of items constantly moving aroundas the items are manipulated or transported by robots. A potentialbenefit is added transparency, both on the supplier side and theconsumer side. On the supplier side, information about current locationsof inventory may be used to avoid overstocking and/or to move items orpallets of items to different locations or warehouses to anticipatedemand. On the consumer side, the information about current locations ofparticular items may be used to determine when a particular package willbe delivered with improved accuracy.

In some examples, some or all of the mobile components 110 withinrobotic fleet 100 may periodically receive charged batteries from abattery exchange station 126 equipped with multiple battery chargers. Inparticular, the station 126 may replace a mobile robot's depletedbatteries with recharged batteries, such that robots might not have towait for batteries to charge. The battery exchange station 126 may beequipped with a robotic manipulator such as a robotic arm. The roboticmanipulator may remove batteries from an individual mobile robot andattach the batteries to available battery chargers. The roboticmanipulator may then move charged batteries located at the station 126into the mobile robot to replace the removed batteries. For instance, anAGV 112 with a weak battery may be controlled to move over to batteryexchange station 126 where a robotic arm pulls a battery out from theAGV 112, puts the battery in a charger, and gives the AGV 112 a freshbattery.

In further examples, battery exchanges may be scheduled by a centralplanning system. For instance, individual mobile robots may beconfigured to monitor their battery charge status. The robots mayperiodically send information to the central planning system indicatingthe status of their batteries. This information may then be used by thecentral planning system to schedule battery replacements for individualrobots within the fleet when needed or convenient.

In some examples, a fleet 100 may contain a number of different types ofmobile components 110 that use different types of batteries. A batteryexchange station 126 may therefore be equipped with different types ofbattery chargers for different types of batteries and/or mobile robots.The battery exchange station 126 may also be equipped with a roboticmanipulator that can replace batteries for different types of robots. Insome examples, mobile robots may have battery containers containingmultiple batteries. For instance, an autonomous fork truck 114 such as apallet jack may have a steel bucket with three or four batteries. Therobotic arm at the station 126 may be configured to lift out the entirebucket of batteries and attach individual batteries to battery chargerson a shelf at the station 126. The robotic arm may then find chargedbatteries to replace the depleted batteries, and move those batteriesback into the bucket before reinserting the bucket into the pallet jack.

In further examples, control system 150 and/or a separate control systemof the battery exchange station 126 may also automate battery managementstrategies. For instance, each battery may have a barcode or otheridentifying mark so that the system can identify individual batteries. Acontrol system of the battery exchange station 126 may count how manytimes individual batteries have been recharged (e.g., to determine whento change water or empty batteries completely). The control system mayalso keep track of which batteries have spent time in which roboticdevices, how long the batteries took to recharge at the station 126 inthe past, and other relevant properties for efficient batterymanagement. This battery usage information may be used by the controlsystem to select batteries for the robotic manipulator to give toparticular mobile robots.

In additional examples, a battery exchange station 126 may also involvea human operator in some cases. For instance, the station 126 couldinclude a rig where people can safely perform manual battery changing ordeliver new batteries to the station for deployment into the fleet 100when necessary.

FIGS. 2A-2D illustrate several examples of robotic devices that may beincluded within a robotic fleet. Other robotic devices which vary inform from those illustrated here as well as other types of roboticdevices may also be included.

FIG. 2A illustrates a robotic truck unloader, according to an exampleembodiment. In some examples, a robotic truck unloader may include oneor more sensors, one or more computers, and one or more robotic arms.The sensors may scan an environment containing one or more objects inorder to capture visual data and/or three-dimensional (3D) depthinformation. Data from the scans may then be integrated into arepresentation of larger areas in order to provide digital environmentreconstruction. In additional examples, the reconstructed environmentmay then be used for identifying objects to pick up, determining pickpositions for objects, and/or planning collision-free trajectories forthe one or more robotic arms and/or a mobile base.

The robotic truck unloader 200 may include a robotic arm 202 with agripping component 204 for gripping objects within the environment. Therobotic arm 202 may use the gripping component 204 to pick up and placeboxes to load or unload trucks or other containers. The truck unloader200 may also include a moveable cart 212 with wheels 214 for locomotion.The wheels 214 may be holonomic wheels that allow the cart 212 to movewith two degrees of freedom. Additionally, a wrap-around front conveyorbelt 210 may be included on the holonomic cart 212. In some examples,the wrap around front conveyer belt may allow the truck loader 200 tounload or load boxes from or to a truck container or pallet withouthaving to rotate gripper 204.

In further examples, a sensing system of robotic truck unloader 200 mayuse one or more sensors attached to a robotic arm 202, such as sensor206 and sensor 208, which may be 2D sensors and/or 3D depth sensors thatsense information about the environment as the robotic arm 202 moves.The sensing system may determine information about the environment thatcan be used by a control system (e.g., a computer running motionplanning software) to pick and move boxes efficiently. The controlsystem could be located on the device or could be in remotecommunication with the device. In further examples, scans from one ormore 2D or 3D sensors with fixed mounts on a mobile base, such as afront navigation sensor 216 and a rear navigation sensor 218, and one ormore sensors mounted on a robotic arm, such as sensor 206 and sensor208, may be integrated to build up a digital model of the environment,including the sides, floor, ceiling, and/or front wall of a truck orother container. Using this information, the control system may causethe mobile base to navigate into a position for unloading or loading. Insome examples, the sensor 208 may include a camera configured to captureimages of the environment of the truck unloader 200 (including otherrobots).

In further examples, the robotic arm 202 may be equipped with a gripper204, such as a digital suction grid gripper. In such embodiments, thegripper may include one or more suction valves that can be turned on oroff either by remote sensing, or single point distance measurementand/or by detecting whether suction is achieved. In additional examples,the digital suction grid gripper may include an articulated extension.In some embodiments, the potential to actuate suction grippers withrheological fluids or powders may enable extra gripping on objects withhigh curvatures.

The truck unloader 200 may additionally include a motor, which may be anelectric motor powered by electrical power, or may be powered by anumber of different energy sources, such as a gas-based fuel or solarpower. Additionally, the motor may be configured to receive power from apower supply. The power supply may provide power to various componentsof the robotic system and could represent, for example, a rechargeablelithium-ion or lead-acid battery. In an example embodiment, one or morebanks of such batteries could be configured to provide electrical power.Other power supply materials and types are also possible.

The truck unloader 200 may include an optical communication interface,such as a display screen 219 a and/or a light source 219 b. In someembodiments, the truck unloader 200 might only include one version ofthe optical communication interface, that is, either the display screen219 a or the light source 219 b. The truck unloader 200 may use itswireless communication interface (not shown) to receive, from thecontrol system 150, data for display by the optical communicationinterface. In some embodiments, the truck unloader 200 may includemultiple optical communication interfaces respectively located onvarious sides of the truck unloader 200 so that the informationdisplayed may be detected or viewed from several angles.

The display screen 219 a may include a liquid crystal display (LCD), aplasma display, or a light emitting diode (LED) display, but otherexamples are possible.

The light source 219 b may include an incandescent light bulb, an LED,or any other light source configured to generate pulses of visible orinfrared light.

74 FIG. 2B illustrates a robotic arm on a pedestal, according to anexample embodiment. More specifically, pedestal robot 220 may bepositioned within an environment such as a warehouse environment andused to pick up, move, and/or otherwise manipulate objects within reach.In some examples, the pedestal robot 220 may be specialized for heavylifting without requiring batteries to operate. The pedestal robot 220may include a robotic arm 222 with an end-effector-mounted gripper 224,which may be of the same type as the robotic manipulator 202 and gripper204 described with respect to the robotic truck unloader 200. Therobotic arm 222 may be mounted on a pedestal 226, which may allow therobotic arm 222 to easily pick up and move nearby packages, such as todistribute packages between different mobile robots. In some examples,the robotic arm 222 may also be operable to construct and/or deconstructpallets of boxes. In further examples, the pedestal 226 may include anactuator to allow a control system to change the height of the roboticarm 222.

In further examples, a bottom surface of the pedestal robot 220 may be apallet-shaped structure. For instance, the bottom surface may havedimension and shape roughly equivalent to other pallets used for objecttransport or storage within a warehouse. By shaping the bottom of thepedestal robot 220 as a pallet, the pedestal robot 220 may be picked upand moved to different locations within a warehouse environment by apallet jack or different type of autonomous fork truck. For instance,when a delivery truck arrives at a particular docking port of thewarehouse, a pedestal robot 220 may be picked up and moved to a locationcloser to the delivery truck to more efficiently process boxes comingfrom or going to the delivery truck.

In additional examples, the pedestal robot 220 may also include one ormore visual sensors to identify boxes and/or other robotic deviceswithin the vicinity of the pedestal robot 220. For instance, a controlsystem of the pedestal robot 220 or a control system may use sensor datafrom sensors on the pedestal robot 220 to identify boxes for the roboticarm 222 and gripper 224 of the pedestal robot 220 to pick up ormanipulate. In further examples, the sensor data may also be used toidentify mobile robotic devices in order to determine where todistribute individual boxes. Other types of robotic fixed manipulationstations may also be used within a heterogeneous robotic fleet as well.

The pedestal robot 220 may include an optical communication interface,such as a display screen 229 a and/or a light source 229 b. In someembodiments, pedestal robot 220 might only include one version of theoptical communication interface, that is, either the display screen 229a or the light source 229 b. The pedestal robot 220 may use its wirelesscommunication interface (not shown) to receive, from the control system150, data for display by the optical communication interface. In someembodiments, the pedestal robot 220 may include multiple opticalcommunication interfaces respectively located on various sides of thepedestal robot 220 so that the information displayed may be detected orviewed from several angles.

The display screen 229 a may include a liquid crystal display (LCD), aplasma display, or a light emitting diode (LED) display, but otherexamples are possible.

The light source 229 b may include an incandescent light bulb, an LED,or any other light source configured to generate pulses of visible orinfrared light.

In some examples, the pedestal robot 220 may include a camera 228configured to capture images of the environment of the pedestal robot220 (including other robots).

FIG. 2C shows an autonomous guided vehicle (AGV), according to anexample embodiment. More specifically, AGV 240 may be a relativelysmall, mobile robotic device that is capable of transporting individualboxes or cases. The AGV 240 may include wheels 242 to allow forlocomotion within a warehouse environment. Additionally, a top surface244 of the AGV 240 may be used to place boxes or other objects fortransport. In some examples, the top surface 244 may include rotatingconveyors to move objects to or from the AGV 240. In additionalexamples, the AGV 240 may be powered by one or more batteries that canbe quickly recharged at a battery charging station and/or exchanged forfresh batteries at a battery exchange station. In further examples, theAGV 240 may additionally include other components not specificallyidentified here, such as sensors for navigation. AGVs with differentshapes and sizes also may be included within a robotic fleet, possiblydepending on the types of packages handled by a warehouse.

The AGV 240 may include an optical communication interface, such as adisplay screen 239 a and/or a light source 239 b. In some embodiments,AGV 240 might only include one version of the optical communicationinterface, that is, either the display screen 239 a or the light source239 b. The AGV 240 may use its wireless communication interface (notshown) to receive, from the control system 150, data for display by theoptical communication interface. In some embodiments, the AGV 240 mayinclude multiple optical communication interfaces respectively locatedon various sides of the AGV 240 so that the information displayed may bedetected or viewed from several angles.

The display screen 239 a may include a liquid crystal display (LCD), aplasma display, or a light emitting diode (LED) display, but otherexamples are possible.

The light source 239 b may include an incandescent light bulb, an LED,or any other light source configured to generate pulses of visible orinfrared light.

In some examples, the AGV 240 may include a camera 238 configured tocapture images of the environment of the AGV 240 (including otherrobots).

FIG. 2D shows an autonomous fork truck, according to an exampleembodiment. More specifically, autonomous fork truck 260 may include aforklift 262 for lifting and/or moving pallets of boxes or other largermaterials. In some examples, the forklift 262 may be elevated to reachdifferent racks of a storage rack or other fixed storage structurewithin a warehouse. The autonomous fork truck 260 may additionallyinclude wheels 264 for locomotion to transport pallets within thewarehouse. In additional examples, the autonomous fork truck may includea motor and power supply as well as a sensing system, such as thosedescribed with respect to robotic truck unloader 200. The autonomousfork truck 260 may also vary in size or shape from the one illustratedin FIG. 2D.

The autonomous fork truck 260 may include an optical communicationinterface, such as a display screen 249 a and/or a light source 249 b.In some embodiments, autonomous fork truck 260 might only include oneversion of the optical communication interface, that is, either thedisplay screen 249 a or the light source 249 b. The autonomous forktruck 260 may use its wireless communication interface (not shown) toreceive, from the control system 150, data for display by the opticalcommunication interface. In some embodiments, the autonomous fork truck260 may include multiple optical communication interfaces respectivelylocated on various sides of the autonomous fork truck 260 so that theinformation displayed may be detected or viewed from several angles.

The display screen 249 a may include a liquid crystal display (LCD), aplasma display, or a light emitting diode (LED) display, but otherexamples are possible.

The light source 249 b may include an incandescent light bulb, an LED,or any other light source configured to generate pulses of visible orinfrared light.

In some examples, the autonomous fork truck 260 may include a camera 248configured to capture images of the environment of the autonomous forktruck 260 (including other robots).

FIGS. 3A, 3B, 3C, 3D, and 3E collectively illustrate example operationsof a robotic fleet in a warehouse, according to an example embodiment.More specifically, a robotic fleet containing different types of robotswith different types of assigned tasks may be deployed within warehouse300. Different robotic devices may operate independently at the sametime according to instructions from a centralized control system or hivemind to complete jobs, such as receiving objects, storing objects,retrieving objects from storage, transporting objects, deliveringobjects from the warehouse, or otherwise processing objects.Additionally, in some examples, two or more robotic devices maycollaborate to perform jobs together, possibly leveraging specializedequipment or functionality of particular devices.

In reference to FIG. 3A, a robotic fleet may include multiple AGV's 302for quickly transporting small totes, such as individual boxes orobjects. The AGV's 302 may be assigned by the centralized control systemto move to particular areas of warehouse 300 to pick up boxes fortransport to another location, such as to store a box or to move a boxto a location to await delivery from the warehouse 300. In someexamples, the AGV's 302 may be assigned to move within an area of reachof a fixed robotic manipulator, such as pedestal robot 304. Morespecifically, pedestal robot 304 may be a robotic arm that is configuredto pick up or otherwise move nearby objects. In some examples, thepedestal robot 304 may be capable of constructing or deconstructingnearby pallets 312 of boxes. In additional examples, the pedestal robot304 may be operable to remove objects from or place particular objectson the AGV's 302 once the AGV's have moved within an area of reach ofthe pedestal robot 304.

In further examples, different types of fixed robotic manipulationstations may be positioned within warehouse 300 as well or instead. Forinstance, instead of using a robotic arm with a gripper, a differenttype of robotic manipulator may be used, possibly depending on the typesof objects stored within warehouse 300, or types of actions needed toprocesses those objects. In some examples, a fixed robotic manipulatormay be configured to open boxes to manipulate items within the boxes aswell. For instance, a warehouse may include a case containing a numberof copies of a consumer product. A robotic manipulator may be capable ofplacing individual copies of the product into smaller boxes (possiblytransported by AGVs) for shipment out of the warehouse.

The robotic fleet may additionally contain other types of mobile roboticdevices for transport of different types or sizes of totes. For example,an autonomous fork truck 306 may be used to pick up and transportpallets, flat support structures upon which boxes may be stacked. Insome examples, storage racks 308 within warehouse 300 may be used tostore pallets of boxes, possibly pallets that are transported to and/orfrom the racks by autonomous fork truck 306. In additional examples,certain pallets 300 may be placed at particular locations within thewarehouse 300 to await further processing. For instance, one of thepallets 300 may be left at a chosen location until a mobile robot isfree to move the pallet, until a pedestal robot 304 is free tomanipulate boxes on the pallet, or until a delivery truck arrives at thewarehouse to transport the pallet to another location outside thewarehouse.

In reference to FIG. 3B, autonomous fork truck 306 may be assigned totransport a particular pallet 314 of boxes to an area within reach ofpedestal robot 304. For instance, the pallet 314 may contain boxes of aparticular type. By transporting the pallet 314 to a location where thepedestal robot 304 can reach it, the pedestal robot 304 may thenredistribute objects from pallet 314 to other areas within reach, suchas onto other pallets 312 or onto one of the nearby AGVs 302 fortransport to other locations.

In some examples, the autonomous fork truck 306 may move to an areawithin reach of the pedestal robot 304 and may then drop off the pallet314 on the ground at a point where the pedestal robot 304 can reach someor all of the objects on the pallet 314. In further examples, afterdropping off the pallet 314, the autonomous fork truck 306 may thenleave the area to perform a different task, such as to retrieve anotherpallet from storage racks 308 or from pallets 310 currently stored onthe ground within warehouse 300. In other examples, the autonomous forktruck 306 may pick up and move a different pallet 312 within reach of304 after dropping off pallet 314, which may be a pallet that waspartially or fully constructed by pedestal robot 304.

In reference to FIG. 3C, pedestal robot 304 may be assigned to transfera box 316 from pallet 314 to AGV 302. Such a process may be repeated forother boxes in pallet 314, perhaps until the boxes of pallet 314 havebeen fully de-palletized. Autonomous fork truck 306 may move back to itsprevious position near other pallets, as shown.

In reference to FIG. 3D, AGV 302 may be assigned to move to an area inproximity to truck 320, thereby transporting box 316 from a locationnear the pedestal robot 304 to a location near truck unloader 318. Then,in reference to FIG. 3E, truck unloader 318 may transfer box 316 fromAGV 302 to truck 320.

Methods 400, 500, and 600 respectively shown in FIG. 4, FIG. 5, and FIG.6 present example methods that can be performed by one or more of (i)the robotic fleet 100 in FIGS. 1A and 1B, (ii) the control system 150 ofFIG. 1B, (iii) the robotic truck unloader 200 in FIGS. 2A and 2E, (iv)the pedestal robot 220 in FIG. 2B, (v) the AGV 240 in FIG. 2C, (vi) theautonomous fork truck 260 in FIG. 2D, and/or (vii) the AGVs 302, thepedestal robot 304, and the autonomous fork truck 306 in FIGS. 3A-3E.The operations may be performed by any combination of one or moresuitable components described herein. FIGS. 4-6 may include one or moreoperations, functions, or actions as illustrated by one or more ofblocks 402-406, 502-508, and 602-610. Although the blocks areillustrated in a sequential order, these blocks may in some instances beperformed in parallel, and/or in a different order than those describedherein. Also, the various blocks may be combined into fewer blocks,divided into additional blocks, and/or removed based upon the desiredimplementation.

Referring to FIG. 4, the method 400 may generally be performed by anycontrol system and/or in interaction with any robot. The control system150 may perform the method 400 in interaction with any robot of therobotic fleet 100, for example.

At block 402, the method 400 includes storing data indicating anassociation between a first optical identifier and a first robot. Forexample, the control system 150 may store, perhaps within a database atnon-transitory computer-readable medium 152, data indicating anassociation between the first optical identifier and any of the AGVs112, autonomous fork trucks 114, truck (un)loaders 116, delivery trucks118, or pedestal robots 122. In some cases, the data may be stored in atable so as to associate a network address of the first robot with dataencoding or representing the first optical identifier. The databasemaintained by the control system 150 may also include data indicatingassociations between other robots and other optical identifiers.

In some instances, the first optical identifier may include atwo-dimensional matrix code, such as a quick response (QR) code or anaugmented reality tag (ARTag). An example first optical identifier 221 atakes the form of a QR code in FIG. 2A. The first optical identifier 221a may be displayed by the display screen 219. In this case, the firstoptical identifier 221 a is recognizable (e.g., with reference to thedatabase stored by control system 150) as being associated with thetruck unloader 200. The first optical identifier 221 a may conveyadditional information about the truck unloader 200 as well. In otherexamples, the first optical identifier 221 a may take the form of anyfiducial marker recognizable as being associated with the truck unloader200. Generally, an optical identifier may include any fiducial marker,symbol, or information that is detectable by an optical sensor such as acamera, light detector, photosensor, photodiode, charge-coupled device,photoresistor, photomultiplier, image sensor, or photodetector, forexample. An optical identifier may be communicated via visible,infrared, and/or ultraviolet light, for example.

In other examples, the first optical identifier may include pulses ofinfrared light and/or pulses of visible light. For instance, the lightsource 219 b may blink or flash intermittently in a manner recognizable(e.g., with reference to the database stored by control system 150) asbeing associated with the truck unloader 200. The first opticalidentifier displayed by the light source 219 b may take the form ofmorse code, for example.

In various examples, the first optical identifier, regardless of formmay be associated with and be displayed by any robot.

At block 404, the method 400 includes sending, to the first robot, dataencoding the first optical identifier for display by the first robot.The first robot may be represented by any of the AGVs 112, autonomousfork trucks 114, truck (un)loaders 116, delivery trucks 118, or pedestalrobots 122. By further example, the control system 150 may use itswireless communication interface to send, to the truck unloader 200,data encoding the first optical identifier for display by either thedisplay screen 219 a or the light source 219 b. That is, the truckunloader 200 may use the data received from the control system 150 todisplay the first optical identifier.

At block 406, the method 400 includes, after sending the data encodingthe first optical identifier, sending, to a second robot, the dataindicating the association between the first optical identifier and thefirst robot. The second robot may be represented by any of the AGVs 112,autonomous fork trucks 114, truck (un)loaders 116, delivery trucks 118,or pedestal robots 122. By further example, after sending the dataencoding the first optical identifier to the truck unloader 200, thecontrol system 150 may use its wireless communication interface to send,to the pedestal robot 220, data indicating the association between thefirst optical identifier and the truck unloader 200. The pedestal robot220 may then use this data to identify the truck unloader 200 within theenvironment of the pedestal robot 220 so that the pedestal robot 220 andthe truck unloader 200 may collaborate to perform a task.

The method 400 may further involve receiving, from the second robot, amessage that includes data encoding the first optical identifier. Inthis context, sending the data indicating the association between thefirst optical identifier and the first robot may include sending thedata indicating the association between the first optical identifier andthe first robot in response to receiving the message. For example, thecontrol system 150 may receive a message from the pedestal robot 220 ofFIG. 2B requesting identification of a robot associated with the firstoptical identifier 221 a that has been detected in an image captured bycamera 228 of the pedestal robot 220. In response, the control system150 may send data to the pedestal robot 220 indicating the associationbetween the first optical identifier 221 a and the truck unloader 200.

The method 400 may further involve determining that a predeterminedamount of time passed since (i) storing the data indicating theassociation between the first optical identifier and the first robot or(ii) sending the data encoding the first optical identifier to the firstrobot.

For example, the control system 150 may generate a timestamp noting thetime when the control system 150 stores the data indicating theassociation between the first optical identifier 221 a and the truckunloader 200. Subsequently, the control system 150 may use an internalor network-based clock to determine that the predetermined amount oftime (e.g., 30 minutes) has passed since storing the data indicating theassociation between the first optical identifier 221 a and the truckunloader 200.

In another example, the control system 150 may generate a timestampnoting the time when the control system 150 sends the data encoding thefirst optical identifier 221 a to the truck unloader 200. Subsequently,the control system 150 may use an internal or network-based clock todetermine that the predetermined amount of time (e.g., 30 minutes) haspassed since sending the the data encoding the first optical identifier221 a to the truck unloader 200.

The method 400 may further involve, in response to determining that thepredetermined amount of time passed, sending, to the first robot, dataencoding a second optical identifier of the first robot for display bythe first robot. For example, after the control system 150 determinesthat the predetermined amount of time passed, the control system 150 maysend, to the truck unloader 200, data encoding a second opticalidentifier 221 b for display by the truck unloader 200. As shown in FIG.2E, the truck unloader 200 may receive the data encoding the secondoptical identifier 221 b and refresh the display screen 219 a to displaythe second optical identifier 221 b. In another example, the secondoptical identifier may include an additional series of pulses of visibleor infrared light for display by the light source 219 b. Periodicallyrefreshing the optical identifier displayed by various robots mayenhance security of the robotic fleet 100, as further described below.

The method 400 may further involve storing data indicating anassociation between the second optical identifier and the first robot.For example, the control system 150 may store new data in its databasethat associates a network address of the truck unloader 200 with dataencoding or representing the second optical identifier 221 b.

The method 400 may further involve identifying a potential securitybreach. For example, the control system 150 may use its camera (or acamera communicatively coupled to the control system 150) to capture animage of an unknown robot in the warehouse. That is, there might not bea record of the unknown robot stored in the database of the controlsystem 150. In recognizing that the unknown robot is foreign to therobotic fleet 100, the control system 150 may identify a potentialsecurity breach.

The method 400 may further involve, in response to identifying thepotential security breach, sending, to the first robot, data encoding asecond optical identifier of the first robot for display by the firstrobot. For example, the control system 150 may send, to the truckunloader 200, data encoding the second optical identifier 221 b fordisplay by the truck unloader 200. As shown in FIG. 2E, the truckunloader 200 may receive the data encoding the second optical identifier221 b and refresh the display screen 219 a to display the second opticalidentifier 221 b. In another example, the second optical identifier mayinclude an additional series of pulses of visible or infrared light fordisplay by the light source 219 b. Refreshing the optical identifierdisplayed by various robots in response to identification of potentialsecurity breaches may prevent an unknown robot from interfering withoperations of the robotic fleet 100.

The method 400 may further involve storing data indicating anassociation between the second optical identifier and the first robot.For example, the control system 150 may store new data in its databasethat associates a network address of the truck unloader 200 with dataencoding or representing the second optical identifier 221 b.

In some examples, the control system 150 may include a camera (or becommunicatively coupled to a camera). In this context, the method 400may further involve capturing, by the camera, an image of the firstoptical identifier displayed by the first robot. For example, thecontrol system 150 may capture an image that includes the first opticalidentifier 221 a displayed by the display screen 219 a of the truckunloader 200.

The method 400 may further involve identifying the first robot bydetecting the first optical identifier within the captured image. Forexample, the control system 150 may use known image processingtechniques to identify the first optical identifier 221 a within theimage, and then identify the truck unloader 200 by using the database toassociate the first optical identifier 221 a with the truck unloader200.

The method 400 may further involve determining a state of the firstrobot, and based on the determined state, sending a message to the firstrobot. For example, the control system 150 may determine that the truckunloader 200 has deviated from its predetermined course because ofnavigational error. In response, the control system 150 may send amessage to the truck unloader 200 that includes the current location ofthe truck unloader 200 and/or instructions for the truck unloader 200 tonavigate back to the predetermined course. In another example, thecontrol system 150 may determine from visual inspection that the truckunloader 200 has fallen over on its side. In this instance, the messagemay include instructions for the truck unloader 200 to power off untilanother robot is available to assist the truck unloader 200.Accordingly, determining the state of the first robot (e.g., truckunloader 200) may include determining the state of the first robot basedon the at least one captured image.

In some examples, the first robot may display an indication of the stateof the first robot. The indication of the state of the first robot maybe included as part of the first optical identifier or may be anindication that is separate and distinct from the first opticalidentifier. The state of the first robot may include operationalconditions such as at least one of (i) an amount of energy stored by abattery of the first robot is less than a threshold amount of energy,(ii) the first robot is experiencing a mechanical failure, and (iii) thefirst robot is experiencing an electrical failure. Other example statesof the first robot are possible. The control system 150 may store dataindicating associations between various displayable indications andstates of robots so that the control system 150 may determine the statesof the robots based on indications displayed by the robots.

In this context, the method 400 may further involve capturing, by acamera, an image of the indication. For example, the camera of thecontrol system 150 may capture an image that includes the first opticalidentifier 221 a. In some examples, the first optical identifier 221 amay indicate a state of the truck unloader 200 as well as identify thetruck unloader 200. In other examples, the truck unloader 200 maydisplay an indication of the state of the truck unloader 200 that isseparate and distinct from the first optical identifier 221 a and thecamera of the control system 150 may capture the separate and distinctindication.

The method 400 may further include identifying the indication within thecaptured image. For instance, the control system 150 may use known imageprocessing techniques to identify the first optical identifier 221 a (oran additional displayed indication) within the captured image. In thiscontext, determining the state of the first robot (e.g., truck unloader200) may include determining the state of the first robot based on theidentified indication.

The method 400 may further involve storing data indicating anassociation between a third optical identifier and a pallet. Forexample, the control system 150 may store data in its database thatassociates pallet 700 of FIG. 7 with a third optical identifier 721 adisplayed by a display screen 719 a. The display screen 719 a may becontrolled by a computing device 725 that is associated with the pallet700. The computing device 725 may include a wireless communicationinterface configured for receiving data from or sending data to thecontrol system 150 or any robots of the robotic fleet 100.

The method 400 may further involve sending, to a computing deviceassociated with the pallet, data encoding the third optical identifierfor display by the computing device. For example, the control system 150may send data encoding the third optical identifier 721 a to thecomputing device 725 so that the computing device 725 may cause thedisplay screen 719 a to display the third optical identifier 721 a. Inanother example, the control system 150 may send data encoding a thirdoptical identifier (e.g., pulses of visible and/or infrared light) fordisplay by the light source 719 b. The light source 719 b may be similarto the light source 219 b of FIG. 2A, for example.

The method 400 may further include, after sending the data encoding thethird optical identifier, sending, to a given robot, the data indicatingthe association between the third optical identifier and the pallet. Forexample, the control system 150 may send, to the autonomous fork truck260 of FIG. 2D, data indicating the association between the thirdoptical identifier 721 a and the pallet 700 so that the autonomous forktruck 260 may identify the pallet 700 by capturing an image of the thirdoptical identifier 721 a. This may help the autonomous fork truck 260 infinding the correct pallet for loading or unloading one or more items.

The method 400 may further involve receiving, from the given robot, amessage that includes data encoding the third optical identifier. Inthis context, sending the data indicating the association between thethird optical identifier and the pallet comprises sending the dataindicating the association between the third optical identifier and thepallet in response to receiving the message.

For example, the control system 150 may receive a message from theautonomous fork truck 260. The message may include a request for thecontrol system 150 to identify the pallet 700 that is displaying thethird optical identifier 721 a. The control system may send, to theautonomous for truck 260, the data indicating the association betweenthe third optical identifier 721 a and the pallet 700 in response toreceiving the message from the autonomous fork truck 260.

In some embodiments, a fixed object within the warehouse 100, such as abattery exchange/charging station 126 (or another sort of dockingstation), a pedestal robot 122, or any other fixed object within theenvironment of the robot may display an optical identifier similar tothe optical identifiers described herein. In such an example, thecontrol system 150 might send data encoding the optical identifier tothe fixed object so that a computing device and a display screencorresponding to the fixed object may display the optical identifier. Inother examples, movable (i.e., non-fixed) objects within the warehouse100 may also display an optical identifier in accordance with anyapplicable functions described herein. A robot moving past the fixedobject may detect the optical identifier displayed by the fixed objectand, based on data received by the robot from the control system 100either before or after detecting the optical identifier, may recognizethe detected optical identifier as a command. The robot may execute afunction in accordance with the command. In other examples, the opticalidentifier may represent information other than a command.

In some examples, the optical identifier may represent a command for arobot to slow down, speed up, dock with the object, move toward theobject, not dock with the object, move away from the object, halt, etc.The optical identifier may represent similar commands for robots aswell.

In an example where the object is a a battery exchange/charging station126, the battery exchange/charging station 126 may display an opticalidentifier indicating whether the battery exchange/charging station 126has a charged battery for exchange with a robot. In an example where abattery of the robot is running low on power, the robot may detect anoptical identifier conveying that the battery exchange/charging station126 does have a charged battery for exchange, and the robot may dockwith the battery exchange/charging station 126 and exchange its drainedbattery for a charged battery. If optical identifier indicates that thebattery exchange/charging station 126 does not have a charged batteryfor exchange, the robot may search for another battery exchange/chargingstation 126 that does have a charged battery for exchange.

In another example, a “signpost” may include a computing device and adisplay screen positioned at an area of the warehouse 100 where robotsoften travel. At times, the display screen may display an opticalidentifier representing robot traffic conditions. If robot traffic nearthe area is heavy, the display screen may display an optical identifierconveying a command for robots in the area to travel at a lower thannormal speed to avoid collisions between the robots. Likewise, if robottraffic near the area is light, the display screen may display anoptical identifier conveying a command for robots in the area to travelat a higher than normal speed. The control system 150 may monitor robottraffic conditions at that area and send data encoding opticalidentifiers representing robot traffic conditions to the signpost fordisplay. Before or after the robot detects such an optical identifier,the control system 150 may send data to the robot conveying the meaningof such optical identifiers so that the robot may react accordingly.

Referring to FIG. 5, the method 500 may generally be performed by anyrobot. Any robot of the robotic fleet 100, for example, may perform themethod 500 in interaction with the the control system 150.

At block 502, the method 500 may include receiving data encoding a firstoptical identifier of a robot. For example, the truck unloader 200 ofFIG. 2A may include a wireless communication interface (not shown). Thewireless communication interface may receive, from the control system150, data encoding (i) the first optical identifier 221 a to bedisplayed by the display screen 219 a or (ii) another first opticalidentifier to be displayed by the light source 219 b. The examplesdescribed above in regard to block 404 of the method 400 may beapplicable to block 502 as well.

At block 504, the method 500 may include displaying, by the opticalcommunication interface, the first optical identifier. For instance, theoptical communication interface of truck unloader 200 may take the formof the display screen 219 a or the light source 219 b. The displayscreen 219 a may display the first optical identifier 221 a or the lightsource 219 b may display a first optical identifier that identifies thetruck unloader 200 in the form of a series of visible or infrared pulsesof light. The optical identifier 221 a may include any type oftwo-dimensional matrix code or any fiducial marker recognizable asidentifying the truck unloader 200.

At block 506, the method 500 may include receiving data encoding asecond optical identifier of the robot. For example, the truck unloader200 may receive data encoding the second optical identifier 221 b oranother second optical identifier that takes the form of a series ofpulses of visible or infrared light.

At block 508, the method 500 may include displaying, by the opticalcommunication interface, the second optical identifier. For example,truck unloader 200 may receive data encoding the second opticalidentifier 221 b for display by the display screen 219 a or may receivedata encoding a second optical identifier for display by the lightsource 219 b.

The method 500 may further involve receiving data encoding a prioritystatus of the robot. For example, the truck unloader 200 may receivedata from the control system 150 indicating a level “1” priority status,which may indicate that the task the truck unloader 200 is performing isat least as important as any other task that another robot isperforming. In another example, the truck unloader 200 may receive datafrom the control system 150 indicating a level “5” priority status,which may indicate that the task the truck unloader is performing is atleast as unimportant as any other task that another robot is performing.Other scales may be used to rate the relative weight of tasks beingperformed by robots as well.

The method 500 may further involve displaying, by the opticalcommunication interface, an indication of the priority status. Forexample, the display screen 219 a of the truck unloader 200 may displaythe indication of the priority status (which may or may not be includedwithin the first optical identifier 221 a). In another example, thelight source 219 b may display the indication of the priority status(which may or may not be included within the first optical identifierdisplayed by the light source 219 b.) In this way, the control system150 or other robots may detect the indication of the priority status andperform particular actions accordingly, as described below.

Referring to FIG. 6, the method 600 may generally be performed by anyrobot. Any robot of the robotic fleet 100, for example, may perform themethod 600 in interaction with any other robot of the robotic fleet 100or the the control system 150.

At block 602, the method 600 may include receiving, from a controlsystem, data encoding a first priority status of the first robot.Referring to FIG. 8 for example, a first truck unloader 200 may receive,from the control system 150, data encoding a first priority status ofthe first truck unloader 200. The first priority status of the firsttruck unloader 200 may reflect the relative importance of the task thefirst truck unloader 200 is to perform on a scale of 1 (highestpriority) to 5 (lowest priority), for example.

At block 604, the method 600 may include capturing an image of anindication of a second priority status of a second robot. In thiscontext, the indication may be displayed by the second robot as theindication is captured. For example, the first truck unloader 200 mayuse camera 208 to capture an image of a second priority status indicator821 a. The second priority status indicator 821 a may be displayed by adisplay screen 819 a of a second truck unloader 800. In another example,the camera 208 may capture an image of a second priority statusindicator that is displayed by the light source 819 b. The secondpriority status indicator displayed by the light source 819 b mayinclude a series of pulses of visible and/or infrared light, asdescribed in examples above. The second priority status of the secondtruck unloader 800 may reflect the relative importance of the task thesecond truck unloader 800 is to perform on a scale of 1 (highestpriority) to 5 (lowest priority), for example.

At block 606, the method 600 may include determining the second prioritystatus by identifying the indication within the captured image. Forexample, the truck unloader 200 may use known image processingtechniques to identify the second priority status indicator 821 a (orthe second priority status indicator displayed by the light source 819b) within the image captured by the camera 208.

At block 608, the method 600 may include comparing the first prioritystatus to the second priority status. In one example, comparing thefirst priority status to the second priority status may includedetermining that the first priority status is higher than the secondpriority status. For instance, the truck unloader 200 may determine thesecond priority status of the second truck unloader 800 to be “3” basedon the second priority status indicator. If the first priority status ofthe first truck unloader 200 is “1,” the first truck unloader 200 maydetermine that the task to be performed by the first truck unloader 200is more important than the task to be performed by the second truckunloader 800.

In another example, comparing the first priority status to the secondpriority status may include determining that the first priority statusis lower than the second priority status. For instance, the truckunloader 200 may determine the second priority status of the secondtruck unloader 800 to be “3” based on the second priority statusindicator. If the first priority status is “5,” the first truck unloader200 may determine that the task to be performed by the first truckunloader 200 less important than the task to be performed by the secondtruck unloader 800.

At block 610, the method 600 may include performing an action based oncomparing the first priority status to the second priority status. Morespecifically, the first truck unloader 200 and the second truck unloader800 may approach each other as they navigate their own respective pathsto complete respective tasks. In the case where the first prioritystatus is higher than the second priority status, the second truckunloader 800 may yield to the first truck unloader 200 so that the firsttruck unloader 200 may move past the second truck unloader 800. In thecase where the first priority status is lower than the second prioritystatus, the first truck unloader 200 may yield to the second truckunloader 800 so that the second truck unloader 800 may move past thefirst truck unloader 200.

In another example, performing the action may include retrieving a firstitem from a third robot after the second robot retrieves a second itemfrom the third robot. In the case where the first priority status ishigher than the second priority status, the first truck unloader 200 mayproceed to load or unload items onto or from the delivery truck 118before the second truck unloader 800 performs such actions. In the casewhere the first priority status is lower than the second prioritystatus, the second truck unloader 800 may proceed to load or unloaditems onto or from the delivery truck 118 before the first truckunloader 200 performs such actions.

In general, robots may communicate with each other by displaying anddetecting optical identifiers. For example a first robot may display anoptical identifier so that a second robot may detect the opticalidentifier. This method of communication may be useful if conventionalwireless or wired communication between the robots fails or experiencesunpredictable latency. For example, when two robots interact to performa task it may be useful to communicate using optical identifiers. Usingoptical identifiers as a means for communication, while perhaps sloweron average than conventional wired or wireless communication, in somecases might be more reliable and/or predictable.

The present disclosure is not to be limited in terms of the particularembodiments described in this application, which are intended asillustrations of various aspects. Many modifications and variations canbe made without departing from its spirit and scope, as will be apparentto those skilled in the art. Functionally equivalent methods andapparatuses within the scope of the disclosure, in addition to thoseenumerated herein, will be apparent to those skilled in the art from theforegoing descriptions. Such modifications and variations are intendedto fall within the scope of the appended claims.

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. In the figures, similar symbols typically identifysimilar components, unless context dictates otherwise. The exampleembodiments described herein and in the figures are not meant to belimiting. Other embodiments can be utilized, and other changes can bemade, without departing from the spirit or scope of the subject matterpresented herein. It will be readily understood that the aspects of thepresent disclosure, as generally described herein, and illustrated inthe figures, can be arranged, substituted, combined, separated, anddesigned in a wide variety of different configurations, all of which areexplicitly contemplated herein.

A block that represents a processing of information may correspond tocircuitry that can be configured to perform the specific logicalfunctions of a herein-described method or technique. Alternatively oradditionally, a block that represents a processing of information maycorrespond to a module, a segment, or a portion of program code(including related data). The program code may include one or moreinstructions executable by a processor for implementing specific logicalfunctions or actions in the method or technique. The program code and/orrelated data may be stored on any type of computer-readable medium suchas a storage device including a disk or hard drive or other storagemedium.

The computer-readable medium may also include non-transitorycomputer-readable media such as computer-readable media that stores datafor short periods of time like register memory, processor cache, andrandom access memory (RAM). The computer-readable media may also includenon-transitory computer-readable media that stores program code and/ordata for longer periods of time, such as secondary or persistent longterm storage, like read only memory (ROM), optical or magnetic disks,compact-disc read only memory (CD-ROM), for example. Thecomputer-readable media may also be any other volatile or non-volatilestorage systems. A computer-readable medium may be considered acomputer-readable storage medium, for example, or a tangible storagedevice.

Moreover, a block that represents one or more information transmissionsmay correspond to information transmissions between software and/orhardware modules in the same physical device. However, other informationtransmissions may be between software modules and/or hardware modules indifferent physical devices.

The particular arrangements shown in the Figures should not be viewed aslimiting. It should be understood that other embodiments can includemore or less of each element shown in a given figure. Further, some ofthe illustrated elements can be combined or omitted. Yet further, anexample embodiment can include elements that are not illustrated in theFigures.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

What is claimed is:
 1. A control system comprising: one or moreprocessors; and a computer-readable storage medium storing instructionsthat, when executed by the one or more processors, cause the controlsystem to perform functions comprising: storing data indicating anassociation between an optical identifier and a pallet; sending, to acomputing device associated with the pallet, data encoding the opticalidentifier for display by the computing device; and after sending thedata encoding the optical identifier, sending, to a robot, the dataindicating the association between the optical identifier and thepallet.
 2. The control system of claim 1, wherein the optical identifiercomprises a two-dimensional matrix code.
 3. The control system of claim1, wherein the optical identifier comprises one or more of (a) pulses ofinfrared light or (b) pulses of visible light.
 4. The control system ofclaim 1, the functions further comprising: receiving, from the robot, amessage that includes data encoding the optical identifier, whereinsending the data indicating the association between the opticalidentifier and the pallet comprises sending the data indicating theassociation between the optical identifier and the pallet in response toreceiving the message.
 5. The control system of claim 1, wherein theoptical identifier is a first optical identifier, the functions furthercomprising: determining that a predetermined amount of time passed since(i) storing the data indicating the association between the firstoptical identifier and the pallet or (ii) sending the data encoding thefirst optical identifier to the pallet; in response to determining thatthe predetermined amount of time passed, sending, to the computingdevice, data encoding a second optical identifier of the pallet fordisplay by the computing device; and storing data indicating anassociation between the second optical identifier and the pallet.
 6. Thecontrol system of claim 1, wherein the optical identifier is a firstoptical identifier, the functions further comprising: identifying apotential security breach; in response to identifying the potentialsecurity breach, sending, to the computing device, data encoding asecond optical identifier of the pallet for display by the computingdevice; and storing data indicating an association between the secondoptical identifier and the pallet.
 7. The control system of claim 1,wherein the data indicating the association between the opticalidentifier and the pallet comprises a network address of the computingdevice.
 8. The control system of claim 1, wherein the functions furthercomprise: capturing, by a camera communicatively coupled to the controlsystem, an image of the optical identifier displayed by the pallet;identifying the pallet by detecting the optical identifier within thecaptured image; determining a state of the pallet; and based on thedetermined state, sending a message to the computing device.
 9. Thecontrol system of claim 8, wherein the computing device displays anindication of the state of the pallet, the functions further comprising:capturing, by the camera, an image of the indication; and identifyingthe indication within the captured image of the indication, whereindetermining the state of the pallet comprises determining the state ofthe pallet based on the identified indication.
 10. The control system ofclaim 8, wherein determining the state of the pallet comprisesdetermining the state of the pallet based on the at least one capturedimage.
 11. A computing device associated with a pallet, the computingdevice comprising: one or more processors; an optical communicationinterface; and a computer-readable storage medium storing instructionsthat, when executed by the one or more processors, cause the computingdevice to perform functions comprising: receiving data encoding a firstoptical identifier of the pallet; displaying, by the opticalcommunication interface, the first optical identifier; receiving dataencoding a second optical identifier of the pallet; and displaying, bythe optical communication interface, the second optical identifier. 12.The computing device of claim 11, wherein the optical communicationinterface comprises a display screen, and wherein the first opticalidentifier comprises a two-dimensional matrix code.
 13. The computingdevice of claim 11, wherein the optical communication interfacecomprises a light source configured to generate one or more of (i)infrared light or (ii) visible light, and wherein the first opticalidentifier comprises one or more of (a) pulses of infrared light or (b)pulses of visible light.
 14. The computing device of claim 11, thefunctions further comprising: receiving data encoding a priority statusof the pallet; and displaying, by the optical communication interface,an indication of the priority status.
 15. A robot comprising: one ormore processors; and a computer-readable storage medium storinginstructions that, when executed by the one or more processors, causethe robot to perform functions comprising: capturing an image of anoptical identifier associated with a pallet, wherein the opticalidentifier is displayed by a computing device associated with thepallet; receiving, from a control system, data indicating theassociation between the optical identifier and the pallet; using thereceived data to identify the pallet; and performing an action inresponse to identifying the pallet.
 16. The robot of claim 15, thefunctions further comprising: sending, to the control system, dataencoding the optical identifier, wherein receiving the data indicatingthe association between the optical identifier and the pallet comprisesreceiving the data indicating the association between the opticalidentifier and the pallet after sending the data encoding the opticalidentifier to the control system.
 17. The robot of claim 16, wherein thedata sent to the control system comprises a request for the controlsystem to provide the data indicating the association between theoptical identifier and the pallet.
 18. The robot of claim 15, whereinthe optical identifier comprises a two-dimensional matrix code.
 19. Therobot of claim 15, wherein the optical identifier comprises one or moreof (a) pulses of infrared light or (b) pulses of visible light.
 20. Therobot of claim 15, wherein the robot is an autonomous fork truck.